Skip to content

branch-4.1: [fix](streaming-job) restore split-bound Java types when reading FE-persisted CDC offset #63219#63272

Merged
yiguolei merged 1 commit into
branch-4.1from
auto-pick-63219-branch-4.1
May 20, 2026
Merged

branch-4.1: [fix](streaming-job) restore split-bound Java types when reading FE-persisted CDC offset #63219#63272
yiguolei merged 1 commit into
branch-4.1from
auto-pick-63219-branch-4.1

Conversation

@github-actions
Copy link
Copy Markdown
Contributor

Cherry-picked from #63219

…ersisted CDC offset (#63219)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

PG streaming jobs with a temporal chunk-key column (e.g. `DATE` PK) fail
during snapshot read with:

```
PSQLException: ERROR: operator does not exist: date <= character varying
  Hint: No operator matches the given name and argument types.
```

**Root cause**

cdc_client reports snapshot splits to FE; FE persists the offset as JSON
via Spring Jackson, where `SqlDateSerializer` writes `java.sql.Date` as
a `"yyyy-MM-dd"` string. When the offset is later read back,
`ObjectMapper.convertValue(offset, SnapshotSplit.class)` lands the bound
values back into `Object[]` as `String` (type info is gone). At read
time `PostgresQueryUtils.readTableSplitDataStatement` calls bare
`statement.setObject(idx, splitEnd[i])`; PG JDBC sends `String` as
VARCHAR oid, and PostgreSQL — strict about operator resolution — refuses
`date <= varchar` (no implicit cast from varchar to date).

The same FE round-trip happens for MySQL too. MySQL server is lenient
enough to implicitly coerce the bound, so the surface error does not
appear, but `SplitKeyUtils.compareObjects` falls back to `toString()`
comparison whenever the restored type doesn't match the Debezium
connect-schema type — a latent issue worth keeping consistent.

**Fix**

Restore the original Java types at the cdc_client side, where the loss
happens:

- `AbstractCdcSourceReader` adds:
- abstract `probeSplitKeyClass(TableId, Column, JobBaseConfig)` —
dialect-specific lookup
- `resolveSplitKeyClass(...)` — per-column cached wrapper (1 probe per
table.column, reused across splits)
- static `convertBounds(Object[], Class<?>, ObjectMapper)` — restores
Object[] elements to the target class
- `PostgresSourceReader` / `MySqlSourceReader` override
`probeSplitKeyClass`: run `SELECT col FROM table WHERE 1=0` and use
`ResultSetMetaData.getColumnClassName(1)` so the JDBC driver itself
decides the Java type. Probe failure throws (no silent fallback — silent
fallback would let the original bug recur).
- `JdbcIncrementalSourceReader.createSnapshotSplit` /
`createStreamSplit` and `MySqlSourceReader.createSnapshotSplit` /
`createBinlogSplit` (4 sites total) apply `convertBounds` before
constructing Flink CDC's `SnapshotSplit` / `FinishedSnapshotSplitInfo`.
- `convertBounds` special-cases `java.sql.Date` / `Timestamp` / `Time`
via `valueOf` to match the JVM-default-TZ semantics of `rs.getObject`
(Jackson's default `SqlDateDeserializer` hard-codes GMT, which would
shift the value in non-UTC TZs). Other types fall through to
`ObjectMapper.convertValue`.

### Release note

Fix PG/MySQL streaming snapshot failing on temporal chunk-key columns
after FE offset JSON round-trip strips the original Java type.
@github-actions github-actions Bot requested a review from yiguolei as a code owner May 15, 2026 01:55
@JNSimba
Copy link
Copy Markdown
Member

JNSimba commented May 19, 2026

run buildall

@yiguolei yiguolei closed this May 20, 2026
@yiguolei yiguolei reopened this May 20, 2026
@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label May 20, 2026
@github-actions
Copy link
Copy Markdown
Contributor Author

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor Author

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit e4d768d into branch-4.1 May 20, 2026
29 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants